# Self-supervised pretraining

voc2vec is a foundational model specifically designed for non-linguistic human data, built on the wav2vec 2.0 framework, with a pretraining dataset covering approximately 125 hours of non-linguistic audio.

RegNetY-32GF feature extraction model, pretrained on 2 billion random web images using SEER method, suitable for image classification and feature extraction tasks

This Vision Transformer model is pretrained using the MSN method and excels in few-shot scenarios, suitable for tasks like image classification

RegNet image classification model trained on ImageNet-1k using self-supervised pretraining and fine-tuning methods

XLM-RoBERTa-XL is a multilingual model pretrained on 2.5TB of filtered CommonCrawl data covering 100 languages, based on an extra-large version of the RoBERTa architecture.

The CORe model is based on the BioBERT architecture, specifically pretrained on clinical records, disease descriptions, and medical literature for predicting in-hospital mortality risk.

BEiT is an image classification model based on Vision Transformer (ViT) architecture, pretrained with self-supervised learning on ImageNet-21k and fine-tuned on ImageNet-1k.